<!DOCTYPE html>

<html>

  <head>
    <title>Robotic Manipulation</title>
    <meta name="Robotic Manipulation" content="text/html; charset=utf-8;" />
    <link rel="canonical" href="http://manipulation.csail.mit.edu/index.html" />

    <script src="https://hypothes.is/embed.js" async></script>
    <script type="text/javascript" src="htmlbook/book.js"></script>

    <script src="htmlbook/mathjax-config.js" defer></script> 
    <script type="text/javascript" id="MathJax-script" defer
      src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-chtml.js">
    </script>
    <script>window.MathJax || document.write('<script type="text/javascript" src="htmlbook/MathJax/es5/tex-chtml.js" defer><\/script>')</script>

    <link rel="stylesheet" href="htmlbook/highlight/styles/default.css">
    <script src="htmlbook/highlight/highlight.pack.js"></script> <!-- http://highlightjs.readthedocs.io/en/latest/css-classes-reference.html#language-names-and-aliases -->
    <script>hljs.initHighlightingOnLoad();</script>

    <link rel="stylesheet" type="text/css" href="htmlbook/book.css" />
  </head>

<body onload="forwardOldChapterLink(); customTags(); MathJax.typeset();">

<div data-type="titlepage">
  <header>
    <h1><a href="index.html" style="text-decoration:none;">Robotic Manipulation</a></h1>
    <p data-type="subtitle">Perception, Planning, and Control</p> 
    <p style="font-size: 18px;"><a href="http://people.csail.mit.edu/russt/">Russ Tedrake</a></p>
    <p style="font-size: 14px; text-align: right;"> 
      &copy; Russ Tedrake, 2020<br/>
      Last modified <span id="last_modified"></span>.</br>
      <script>
      var d = new Date(document.lastModified);
      document.getElementById("last_modified").innerHTML = d.getFullYear() + "-" + (d.getMonth()+1) + "-" + d.getDate();</script>
      <a href="misc.html">How to cite these notes, use annotations, and give feedback.</a><br/>
    </p>
  </header>
</div>

<p><b>Note:</b> These are working notes used for <a
href="http://manipulation.csail.mit.edu/Fall2020/">a course being taught
at MIT</a>. They will be updated throughout the Fall 2020 semester.  <!-- <a 
href="https://www.youtube.com/channel/UChfUOAhz7ynELF-s_1LPpWg">Lecture  videos are available on YouTube</a>.--></p> 

<section id="search"><h1>Search these notes</h1>

  <!--
  <form name="mysearch" action="http://www.google.com/search" method="get" onSubmit="document.getElementById('hiddenquery').value='site:manipulation.csail.mit.edu ' + this.qfront.value">
    <p>Search these notes: &nbsp; &nbsp;
    <input id="hiddenquery" type="hidden" name="q" />
    <input name="qfront" type="text" style="width: 200px" value="" />
    <input type="submit" value="Search" /><br />
    </div>
  </form>
  -->

  <script async src="https://cse.google.com/cse.js?cx=8aec585254dc603b9"></script>
  <div class="gcse-search"></div>  

</section>

<section id="table_of_contents">
<h1>Table of Contents</h1>
<ul>
  <li><a href="#preface">Preface</a></li>
  <li><a href="intro.html">Chapter 1: Introduction</a></li>
  <ul>
    <li><a href=intro.html#section1>Manipulation is more than pick-and-place</a></li>
    <li><a href=intro.html#section2>Open-world manipulation</a></li>
    <li><a href=intro.html#section3>Simulation</a></li>
    <li><a href=intro.html#section4>These notes are interactive</a></li>
    <li><a href=intro.html#section5>Model-based design and analysis</a></li>
    <li><a href=intro.html#section6>Organization of these notes</a></li>
  </ul>
  <li><a href="robot.html">Chapter 2: Let's get you a robot</a></li>
  <ul>
    <li><a href=robot.html#section1>Arms</a></li>
    <ul>
      <li>Position-controlled robots</li>
      <li>Torque-controlled robots</li>
      <li>A proliferation of hardware</li>
      <li>Simulating the Kuka iiwa</li>
    </ul>
    <li><a href=robot.html#section2>Hands</a></li>
    <ul>
      <li>Dexterous hands</li>
      <li>Simple grippers</li>
      <li>Soft/underactuated hands</li>
      <li>Other end effectors</li>
      <li>If you haven't seen it...</li>
    </ul>
    <li><a href=robot.html#section3>Sensors</a></li>
    <li><a href=robot.html#section4>Putting it all together</a></li>
    <li><a href=robot.html#section5>Exercises</a></li>
  </ul>
  <li><a href="pick.html">Chapter 3: Basic Pick and Place</a></li>
  <ul>
    <li><a href=pick.html#section1>Monogram Notation</a></li>
    <li><a href=pick.html#section2>Pick and place via spatial transforms</a></li>
    <li><a href=pick.html#section3>Spatial Algebra</a></li>
    <li><a href=pick.html#kinematics>Forward kinematics</a></li>
    <ul>
      <li>The kinematic tree</li>
      <li>Forward kinematics for pick and place</li>
    </ul>
    <li><a href=pick.html#jacobian>Differential kinematics (Jacobians)</a></li>
    <li><a href=pick.html#section6>Differential inverse kinematics</a></li>
    <ul>
      <li>The Jacobian pseudo-inverse</li>
      <li>Invertibility of the Jacobian</li>
    </ul>
    <li><a href=pick.html#section7>Defining the grasp and pre-grasp poses</a></li>
    <li><a href=pick.html#section8>A pick and place trajectory</a></li>
    <li><a href=pick.html#section9>Putting it all together</a></li>
    <li><a href=pick.html#diff_ik_w_constraints>Differential inverse kinematics with
  constraints</a></li>
    <ul>
      <li>Pseudo-inverse as an optimization</li>
      <li>Adding velocity constraints</li>
      <li>Adding position and acceleration constraints</li>
      <li>Joint centering</li>
      <li>Alternative formulations</li>
    </ul>
    <li><a href=pick.html#section11>Exercises</a></li>
  </ul>
  <li><a href="pose.html">Chapter 4: Geometric Pose Estimation</a></li>
  <ul>
    <li><a href=pose.html#section1>Cameras and depth sensors</a></li>
    <ul>
      <li>Depth sensors</li>
      <li>Simulation</li>
    </ul>
    <li><a href=pose.html#section2>Representations for geometry</a></li>
    <li><a href=pose.html#section3>Point cloud registration with known correspondences</a></li>
    <li><a href=pose.html#section4>Iterative Closest Point (ICP)</a></li>
    <li><a href=pose.html#section5>Dealing with partial views and outliers</a></li>
    <ul>
      <li>Detecting outliers</li>
      <li>Point cloud segmentation</li>
      <li>Generalizing correspondence</li>
      <li>Soft correspondences</li>
      <li>Nonlinear optimization</li>
      <li>Global optimization</li>
    </ul>
    <li><a href=pose.html#section6>Non-penetration and "free-space" constraints</a></li>
    <ul>
      <li>Free space constraints as non-penetration constraints</li>
    </ul>
    <li><a href=pose.html#section7>Putting it all together</a></li>
    <li><a href=pose.html#section8>Looking ahead</a></li>
    <li><a href=pose.html#exercises>Exercises</a></li>
  </ul>
  <li><a href="clutter.html">Chapter 5: Bin Picking</a></li>
  <ul>
    <li><a href=clutter.html#section1>Generating random cluttered scenes</a></li>
    <ul>
      <li>Falling things</li>
      <li>Static equilibrium with frictional contact</li>
      <li>A few of the nuances of simulating contact</li>
      <li>Static equilibrium as an optimization constraint</li>
    </ul>
    <li><a href=clutter.html#section2>Grasp selection</a></li>
    <ul>
      <li>Point cloud pre-processing</li>
      <li>Estimating normals and local curvature</li>
      <li>Evaluating a candidate grasp</li>
      <li>Generating grasp candidates</li>
    </ul>
    <li><a href=clutter.html#section3>The corner cases</a></li>
    <li><a href=clutter.html#section4>Putting it all together</a></li>
    <ul>
      <li>A simple state machine</li>
    </ul>
    <li><a href=clutter.html#exercises>Exercises</a></li>
  </ul>
  <li><a href="segmentation.html">Chapter 6: Object detection and
segmentation</a></li>
  <ul>
    <li><a href=segmentation.html#section1>Getting to big data</a></li>
    <ul>
      <li>Crowd-sourced annotation datasets</li>
      <li>Segmenting new classes via fine tuning</li>
      <li>Annotation tools for manipulation</li>
      <li>Synthetic datasets</li>
    </ul>
    <li><a href=segmentation.html#section2>Object detection and segmentation</a></li>
    <li><a href=segmentation.html#section3>Putting it all together</a></li>
    <li><a href=segmentation.html#section4>Exercises</a></li>
  </ul>
  <li><a href="force.html">Chapter 7: Force Control</a></li>
  <ul>
    <li><a href=force.html#section1>A simple model</a></li>
    <ul>
      <li>(Direct) force control</li>
      <li>A force-based "flip-up" strategy</li>
      <li>Indirect force control</li>
      <li>A stiffness-control-based "flip-up" strategy</li>
      <li>Hybrid position/force control</li>
    </ul>
    <li><a href=force.html#section2>Peg in hole</a></li>
    <li><a href=force.html#section3>Manipulator control</a></li>
    <ul>
      <li>Joint stiffness control</li>
      <li>Cartesian stiffness control</li>
      <li>Some implementation details on the iiwa</li>
    </ul>
    <li><a href=force.html#section4>Putting it all together</a></li>
    <li><a href=force.html#section5>Exercises</a></li>
  </ul>
  <li><a href="trajectories.html">Chapter 8: Motion Planning</a></li>
  <ul>
    <li><a href=trajectories.html#section1>Inverse Kinematics</a></li>
    <ul>
      <li>Closed-form solutions</li>
      <li>IK as constrained optimization</li>
      <li>Grasp planning as IK</li>
    </ul>
    <li><a href=trajectories.html#section2>Kinematic trajectory optimization</a></li>
    <li><a href=trajectories.html#section3>Sample-based motion planning</a></li>
    <li><a href=trajectories.html#section4>Exercises</a></li>
  </ul>
  <li><a href="rl.html">Chapter 9: Reinforcement Learning</a></li>
  <ul>
    <li><a href=rl.html#section1>Exercises</a></li>
  </ul>
<p style="margin-bottom: 0; text-decoration: underline;font-variant: small-caps;"><b>Appendix</b></p>
  <li><a href="drake.html">Appendix A: Drake</a></li>
  <ul>
    <li><a href=drake.html#section1>Pydrake</a></li>
    <li><a href=drake.html#section2>Online Jupyter notebooks</a></li>
    <li><a href=drake.html#section3>Running on your own machine</a></li>
    <ul>
      <li>Install Drake</li>
    </ul>
    <li><a href=drake.html#section4>Getting help</a></li>
  </ul>
  <li><a href="station.html">Appendix B: Setting up your own "Manipulation Station"</a></li>
  <ul>
    <li><a href=station.html#section1>Message Passing</a></li>
    <li><a href=station.html#section2>Kuka LBR iiwa + Schunk WSG Gripper</a></li>
    <li><a href=station.html#section3>Intel Realsense D415 Depth Cameras</a></li>
    <li><a href=station.html#section4>Miscellaneous hardware.</a></li>
  </ul>
  <li><a href="misc.html">Appendix C: Miscellaneous</a></li>
  <ul>
    <li><a href=misc.html#cite>How to cite these notes</a></li>
    <li><a href=misc.html#annotation>Annotation tool etiquette</a></li>
    <li><a href=misc.html#feedback>Please give me feedback!</a></li>
  </ul>
</ul>
</section>


<section id="preface"><h1>Preface</h1>

  <p>I've always loved robots, but it's only relatively recently that I've
  turned my attention to robotic manipulation.  I particularly like the
  challenge of building robots that can master physics to achieve
  human/animal-like dexterity and agility.  It was <a
  href="http://underactuated.mit.edu/intro.html">passive
  dynamic walkers</a> and the beautiful analysis that accompanies them that
  first helped cement this centrality of dynamics in my view of the world and my
  approach to robotics.  From there I became fascinated with (experimental)
  fluid dynamics, and the idea that birds with articulated wings actually
  "manipulate" the air to achieve incredible efficiency and agility. Humanoid
  robots and fast-flying aerial vehicles in clutter forced me to start thinking
  more deeply about the role of perception in dynamics and control. Now I
  believe that this interplay between perception and dynamics is truly
  fundamental, and I am passionate about the observation that relatively
  "simple" problems in manipulation (how do I button up my dress shirt?) expose
  the problem beautifully.</p>
  
  <p>My approach to programming robots has always been very
  computational/algorithmic.  I started out using tools primarily from machine
  learning (especially reinforcement learning) to develop the control systems
  for simple walking machines; but as the robots and tasks got more complex I
  turned to more sophisticated tools from model-based planning and
  optimization-based control.  In my view, no other discipline has thought so
  deeply about dynamics as has control theory, and the algorithmic efficiency
  and guaranteed performance/robustness that can be obtained by the best
  model-based control algorithms far surpasses what we can do today with
  learning control.  Unfortunately, the mathematical maturity of
  controls-related research has also led the field to be relatively conservative
  in their assumptions and problem formulations; the requirements for robotic
  manipulation break these assumptions.  For example, robust control typically
  assumes dynamics that are (nearly) smooth and uncertainty that can be
  represented by simple distributions or simple sets; but in robotic
  manipulation, we must deal with the non-smooth mechanics of contact and
  uncertainty that comes from varied lighting conditions, and different numbers
  of objects with unknown geometry and dynamics.  In practice, no
  state-of-the-art robotic manipulation system to date (that I know of) uses
  rigorous control theory to design even the low-level feedback that determines
  when a robot makes and breaks contact with the objects it is manipulating.  An
  explicit goal of these notes is to try to change that.</p>
  
  <p>In the past few years, deep learning has had an unquestionable impact on
  robotic perception, unblocking some of the most daunting challenges in
  performing manipulation outside of a laboratory or factory environment.  We
  will discuss relevant tools from deep learning for object recognition,
  segmentation, pose/keypoint estimation, shape completion, etc.  Now relatively
  old approaches to learning control are also enjoying an incredible surge in
  popularity, fueled in part by massive computing power and increasingly
  available robot hardware and simulators.  Unlike learning for perception,
  learning control algorithms are still far from a technology, with some of the
  most impressive looking results still being hard to understand and to
  reproduce.  But the recent work in this area has unquestionably highlighted
  the pitfalls of the conservatism taken by the controls community. Learning
  researchers are boldly formulating much more aggressive and exciting problems
  for robotic manipulation than we have seen before -- in many cases we are
  realizing that some manipulation tasks are actually quite easy, but in other
  cases we are finding problems that are still fundamentally hard.</p>
  
  <p>Finally, it feels that the time is ripe for robotic manipulation to have a
  real and dramatic impact in the world, in fields from logistics to home
  robots.  Over the last few years, we've seen UAVs/drones transition from
  academic curiosities into consumer products.  Even more recently, autonomous
  driving has transitioned from academic research to industry, at least in
  terms of dollars invested.  Manipulation feels like the next big thing that
  will make the leap from robotic research to practice. It's still a bit risky
  for a venture capitalist to invest in, but nobody doubts the size of the
  market once we have the technology.  How lucky are we to potentially be able
  to play a role in that transition?</p>
  
  <p>So this is where the notes begin... we are at an incredible crossroads
  between learning and control and robotics with an opportunity to have
  immediate impact in industrial and consumer applications and potentially even
  to forge entirely new eras for systems theory and controls.  I'm just trying
  to hold on and to enjoy the ride.</p>
  
  <section><h1>A manipulation toolbox</h1>
  
    <p>Another explicit goal of these lecture notes is to provide high-quality
    implementations of the most useful tools in a manipulation scientist's
    toolbox.  When I am forced to choose between mathematical clarity and
    runtime performance, the clear formulation is always my first priority; I
    will try to include a performant formulation, too, if possible or try to
    give pointers to alternatives.  Manipulation research is moving quickly, and
    I aim to evolve these notes to keep pace.  I hope that the software
    components provided in
    <drake></drake> and in these notes can be directly useful to you in your own
    work.</p>
  
    <p>If you would like to replicate any or all of the hardware that we use for
    these notes, you can find information and instructions in the <a
    href="station.html">appendix</a>.</p>
  
    <p>As you use the code, please consider <a
    href="https://drake.mit.edu/getting_help.html">contributing back</a>
    (especially to the mature code in <drake></drake>).  Even questions/bug
    reports can be important contributions.  If you have questions/find issues
    with these notes, please submit them <a
    href="https://github.com/RussTedrake/manipulation/issues">here</a>.</p>
  
  </section>

  <p style="text-align:right;"><a href="intro.html">First chapter</a></p>

</section> <!-- end preface -->

<div id="footer">
<hr/>
<table style="width:100%;">
  <tr><td><a href="https://accessibility.mit.edu/">Accessibility</a></td><td style="text-align:right">&copy; Russ
    Tedrake, 2020</td></tr>
</table>
</div>


</body>
</html>
